Génération automatique de questionnaires à choix multiples pédagogiques : évaluation de l'homogénéité des options

Recent years have seen a revival of Intelligent Tutoring Systems. In order to make these systems widely usable by teachers and learners, they have to provide means to assist teachers in their task of exercise generation. Among these exercises, multiple-choice tests are very common. However, writing Multiple-Choice Questions (MCQ) that correctly assess a learner's level is a complex task. Guidelines were developed to manually write MCQs, but an automatic evaluation of MCQ quality would be a useful tool for teachers.We are interested in automatic evaluation of distractor (wrong answer choice) quality. To do this, we studied characteristics of relevant distractors from multiple-choice test writing guidelines. This study led us to assume that homogeneity between distractors and answer is an important criterion to validate distractors. Homogeneity is both syntactic and semantic. We validated the definition of homogeneity by a MCQ corpus analysis, and we proposed methods for automatic recognition of syntactic and semantic homogeneity based on this analysis.Then, we focused our work on distractor semantic homogeneity. To automatically estimate it, we proposed a ranking model by machine learning, combining different semantic homogeneity measures. The evaluation of the model showed that our method is more efficient than existing work to estimate distractor semantic homogeneityCes dernières années ont connu un renouveau des Environnements Informatiques pour l'Apprentissage Humain. Afin que ces environnements soient largement utilisés par les enseignants et les apprenants, ils doivent fournir des moyens pour assister les enseignants dans leur tâche de génération d'exercices. Parmi ces exercices, les Questionnaires à Choix Multiples (QCM) sont très présents. Cependant, la rédaction d'items à choix multiples évaluant correctement le niveau d'apprentissage des apprenants est une tâche complexe. Des consignes ont été développées pour rédiger manuellement des items, mais une évaluation automatique de la qualité des items constituerait un outil pratique pour les enseignants.Nous nous sommes intéressés à l'évaluation automatique de la qualité des distracteurs (mauvais choix de réponse). Pour cela, nous avons étudié les caractéristiques des distracteurs pertinents à partir de consignes de rédaction de QCM. Cette étude nous a conduits à considérer que l'homogénéité des distracteurs et de la réponse est un critère important pour valider les distracteurs. L'homogénéité est d'ordre syntaxique et sémantique. Nous avons validé la définition de l'homogénéité par une analyse de corpus de QCM, et nous avons proposé des méthodes de reconnaissance automatique de l'homogénéité syntaxique et sémantique à partir de cette analyse.Nous nous sommes ensuite focalisé sur l'homogénéité sémantique des distracteurs. Pour l'estimer automatiquement, nous avons proposé un modèle d'ordonnancement par apprentissage, combinant différentes mesures d'homogénéité sémantique. L'évaluation du modèle a montré que notre méthode est plus efficace que les travaux existants pour estimer l'homogénéité sémantique des distracteurs

Réponse à des tests de compréhension.

National audienceDans cet article, nous présentons une adaptation d’un système de questions-réponses existant pour une tâche de réponse à des questions de compréhension de textes. La méthode proposée pour sélectionner les réponses correctes repose sur la reconnaissance d’implication textuelle entre les hypothèses et les textes. Les spécificités de cette méthode sont la génération d’hypothèses par réécriture syntaxique, et l’évaluation de plusieurs critères de distance,adaptés pour gérer des variantes de termes

Multiple Choice Question Corpus Analysis for Distractor Characterization

International audienceIn this paper, we present a study of MCQ aiming to define criteria in order to automatically select distractors. We are aiming to show that distractor editing follows rules like syntactic and semantic homogeneity according to associated answer, and the possibility to automatically identify this homogeneity. Manual analysis shows that homogeneity rule is respected to edit distractors and automatic analysis shows the possibility to reproduce these criteria. These ones can be used in future works to automatically select distractors, with the combination of other criteria

Selecting answers with structured lexical expansion and discourse relations: LIMSI's participation at QA4MRE 2013

International audiencen this paper, we present the LIMSI’s participation to QA4MRE2013. We decided to test two kinds of methods. The first one focuses on complex questions, such as causal questions, and exploits discourse relations. Relation recognition shows promising results, however it has to be improved to have an impact on answer selection. The second method is based on semantic variations. We explored the English Wiktionary to find reformulations of words in the definitions, and used these reformulations to index the documents and select passages in the Entrance exams task

Adaptation of LIMSI's QALC for QA4MRE.

International audienceIn this paper, we present LIMSI participation to one of the pilot tasks of QA4MRE at CLEF 2012: Machine Reading of Biomedical Texts about Alzheimer. For this exercise, we adapted an existing question answering (QA) system, QALC, by searching answers in the reading document. This basic version was used for the evaluation and obtains 0.2, which was increased to 0.325 after basic corrections. We developed then different methods for choosing an answer, based on the expected answer type and the question plus answer rewritten to form hypothesis compared with candidates sentences. We also conducted studies on relation extraction by using an existing system. The last version of our system obtains 0.375

Automatic generation of educational multiple-choice questions : evaluation of option homogeneity

Ces dernières années ont connu un renouveau des Environnements Informatiques pour l'Apprentissage Humain. Afin que ces environnements soient largement utilisés par les enseignants et les apprenants, ils doivent fournir des moyens pour assister les enseignants dans leur tâche de génération d'exercices. Parmi ces exercices, les Questionnaires à Choix Multiples (QCM) sont très présents. Cependant, la rédaction d'items à choix multiples évaluant correctement le niveau d'apprentissage des apprenants est une tâche complexe. Des consignes ont été développées pour rédiger manuellement des items, mais une évaluation automatique de la qualité des items constituerait un outil pratique pour les enseignants.Nous nous sommes intéressés à l'évaluation automatique de la qualité des distracteurs (mauvais choix de réponse). Pour cela, nous avons étudié les caractéristiques des distracteurs pertinents à partir de consignes de rédaction de QCM. Cette étude nous a conduits à considérer que l'homogénéité des distracteurs et de la réponse est un critère important pour valider les distracteurs. L'homogénéité est d'ordre syntaxique et sémantique. Nous avons validé la définition de l'homogénéité par une analyse de corpus de QCM, et nous avons proposé des méthodes de reconnaissance automatique de l'homogénéité syntaxique et sémantique à partir de cette analyse.Nous nous sommes ensuite focalisé sur l'homogénéité sémantique des distracteurs. Pour l'estimer automatiquement, nous avons proposé un modèle d'ordonnancement par apprentissage, combinant différentes mesures d'homogénéité sémantique. L'évaluation du modèle a montré que notre méthode est plus efficace que les travaux existants pour estimer l'homogénéité sémantique des distracteurs.Recent years have seen a revival of Intelligent Tutoring Systems. In order to make these systems widely usable by teachers and learners, they have to provide means to assist teachers in their task of exercise generation. Among these exercises, multiple-choice tests are very common. However, writing Multiple-Choice Questions (MCQ) that correctly assess a learner's level is a complex task. Guidelines were developed to manually write MCQs, but an automatic evaluation of MCQ quality would be a useful tool for teachers.We are interested in automatic evaluation of distractor (wrong answer choice) quality. To do this, we studied characteristics of relevant distractors from multiple-choice test writing guidelines. This study led us to assume that homogeneity between distractors and answer is an important criterion to validate distractors. Homogeneity is both syntactic and semantic. We validated the definition of homogeneity by a MCQ corpus analysis, and we proposed methods for automatic recognition of syntactic and semantic homogeneity based on this analysis.Then, we focused our work on distractor semantic homogeneity. To automatically estimate it, we proposed a ranking model by machine learning, combining different semantic homogeneity measures. The evaluation of the model showed that our method is more efficient than existing work to estimate distractor semantic homogeneit

Load-carrying capacity of ultra-thin shells with and without CNTs reinforcement

Isotropic ultra-thin shells or membranes, as well as cable–membrane structures, cannot resist loads at the initial state and always require a form-finding process to reach the steady state. After this stage, they can work in a pure membrane state and quickly experience large deflection behavior, even with a small amplitude of load. This paper aims to improve the load-carrying capacity and strength of membrane structures via exploiting the advantages of functionally graded carbon-nanotube-reinforced composite (FG-CNTRC) material. In this work, the load-carrying capacity and nonlinear behavior of membrane structures with and without CNTs reinforcement are first investigated using a unified adaptive approach (UAA). As an advantage of UAA, both form finding and postbuckling analysis are performed conveniently and simultaneously based on a modified Riks method. Different from the classical membrane theory, the present theory (first-order shear deformation theory) simultaneously takes into account the membrane, shear and bending strains/stiffnesses of structures. Accordingly, the present formulation can be applied adaptively and naturally to various types of FG-CNTRC structures: plates, shells and membranes. A verification study is conducted to show the high accuracy of the present approach and formulation. Effects of CNTs distribution, volume fraction, thickness, curvature, radius-to-thickness and length-to-radius ratios on the form-finding and postbuckling behavior of FG-CNTRC membranes are particularly investigated. In particular, equilibrium paths of FG-CNTRC membrane structures are first provided in this paper.Published versionThis research was funded by National Research Foundation of Korea grant number NRF2020R1A4A2002855

Load-Carrying Capacity of Ultra-Thin Shells with and without CNTs Reinforcement

Isotropic ultra-thin shells or membranes, as well as cable–membrane structures, cannot resist loads at the initial state and always require a form-finding process to reach the steady state. After this stage, they can work in a pure membrane state and quickly experience large deflection behavior, even with a small amplitude of load. This paper aims to improve the load-carrying capacity and strength of membrane structures via exploiting the advantages of functionally graded carbon-nanotube-reinforced composite (FG-CNTRC) material. In this work, the load-carrying capacity and nonlinear behavior of membrane structures with and without CNTs reinforcement are first investigated using a unified adaptive approach (UAA). As an advantage of UAA, both form finding and postbuckling analysis are performed conveniently and simultaneously based on a modified Riks method. Different from the classical membrane theory, the present theory (first-order shear deformation theory) simultaneously takes into account the membrane, shear and bending strains/stiffnesses of structures. Accordingly, the present formulation can be applied adaptively and naturally to various types of FG-CNTRC structures: plates, shells and membranes. A verification study is conducted to show the high accuracy of the present approach and formulation. Effects of CNTs distribution, volume fraction, thickness, curvature, radius-to-thickness and length-to-radius ratios on the form-finding and postbuckling behavior of FG-CNTRC membranes are particularly investigated. In particular, equilibrium paths of FG-CNTRC membrane structures are first provided in this paper

Multiple Choice Question Corpus Analysis for Distractor Characterization

In this paper, we present a study of MCQ aiming to define criteria in order to automatically select distractors. We are aiming to show that distractor editing follows rules like syntactic and semantic homogeneity according to associated answer, and the possibility to automatically identify this homogeneity. Manual analysis shows that homogeneity rule is respected to edit distractors and automatic analysis shows the possibility to reproduce these criteria. These ones can be used in future works to automatically select distractors, with the combination of other criteria